When Is Word Sense Disambiguation Difficult? A Crowdsourcing Approach

We identified features that drive differential accuracy in word sense disambiguation (WSD) by building regression models using 10,000 coarse-grained WSD instances which were labeled on Mturk. Features predictive of accuracy include properties of the target word (word frequency, part of speech, and number of possible senses), the example context (length), and the Turker’s engagement with our task. The resulting model gives insight into which words are difficult to disambiguate. We also show that having many Turkers label the same instance provides at least a partial substitute for more expensive annotation. Disciplines Business This working paper is available at ScholarlyCommons: http://repository.upenn.edu/wharton_research_scholars/116 When is Word Sense Disambiguation Difficult? A Crowdsourcing Approach Adam Kapelner Krishna Kaliannan The Wharton School of the University of Pennyslvania Department of Statistics 3730 Walnut Street Philadelphia, PA 19104 {kapelner, kkali, foster}@wharton.upenn.edu Dean Foster Lyle Ungar University of Pennyslvania Department of Computer Science 200 S. 33rd St 504 Levine Philadelphia, PA 19104 [email protected]